Information processing apparatus, information processing method, and program

ABSTRACT

There is provided an information processing apparatus, an information processing method, and a program capable of further diversifying a method of implementing a virtual object using the AR technology, the information processing apparatus including: a display control unit configured to perform a first process of displaying a virtual object superimposed on a real space at a position which is based on a real object detected in the real space or a second process of displaying the virtual object superimposed on the real space at a position which is not based on the real object detected in the real space.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/577,113 (filed on Nov. 27, 2017), which is a National Stage PatentApplication of PCT International Patent Application No.PCT/JP2016/056666 (filed on Mar. 3, 2016) under 35 U.S.C. § 371, whichclaims priority to Japanese Patent Application No. 2015-120186 (filed onJun. 15, 2015), which are all hereby incorporated by reference in theirentirety.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a program.

BACKGROUND ART

In recent years, technology called augmented reality (AR) that presentsa user with additional information superimposed on the real world hasattracted public attention. In the AR technology, information presentedto the user is also called annotation and can be visualized usingvarious forms of virtual objects such as text, icons, or animation. Forexample, a technique in which a virtual object of the AR is displayed inassociation with a real object, for example, a virtual object isdisplayed on a surface of a real object, or a virtual object is causedto float above a real object and be displayed is disclosed in PatentLiterature 1.

CITATION LIST Patent Literature

Patent Literature 1: WO 2014/162823

DISCLOSURE OF INVENTION Technical Problem

It has not been long since the AR technology proposed in the PatentLiterature 1 or the like was developed, and it is difficult to say thattechniques for utilizing the AR in various situations have beensufficiently proposed. For example, diversification of a method ofimplementing a virtual object using the AR technology is also one oftechniques which are not sufficiently proposed.

In this regard, the present disclosure proposes an informationprocessing apparatus, an information processing method, and a program,which are novel and improved and capable of further diversifying amethod of implementing a virtual object using the AR technology.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing apparatus including: a display control unit configured toperform a first process of displaying a virtual object superimposed on areal space at a position which is based on a real object detected in thereal space or a second process of displaying the virtual objectsuperimposed on the real space at a position which is not based on thereal object detected in the real space.

Further, according to the present disclosure, there is provided aninformation processing method including: performing, by a processor, afirst process of displaying a virtual object superimposed on a realspace at a position which is based on a real object detected in the realspace or a second process of displaying the virtual object superimposedon the real space at a position which is not based on the real objectdetected in the real space.

Further, according to the present disclosure, there is provided aprogram causing a computer to function as: a display control unitconfigured to perform a first process of displaying a virtual objectsuperimposed on a real space at a position which is based on a realobject detected in the real space or a second process of displaying thevirtual object superimposed on the real space at a position which is notbased on the real object detected in the real space.

Advantageous Effects of Invention

As described above, according to the present disclosure, it is possibleto further diversify methods of implementing a virtual object using theAR technology. Note that the effects described above are not necessarilylimitative. With or in the place of the above effects, there may beachieved any one of the effects described in this specification or othereffects that may be grasped from this specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an example of an external configurationof a display device according to the present embodiment.

FIG. 2 is a block diagram illustrating an example of an internalconfiguration of a display device according to the present embodiment.

FIG. 3 is a diagram illustrating an overview of a process executed in adisplay device according to the present embodiment.

FIG. 4 is a diagram for describing a display control process performedby a display device according to the present embodiment.

FIG. 5 is a diagram for describing a display example of a virtual objectaccording to the present embodiment.

FIG. 6 is a diagram for describing a display example of a virtual objectaccording to the present embodiment.

FIG. 7 is a diagram for describing a display example of a virtual objectaccording to the present embodiment.

FIG. 8 is a diagram for describing a display example of a virtual objectaccording to the present embodiment.

FIG. 9 is a diagram for describing a display example of a virtual objectaccording to the present embodiment.

FIG. 10 is a diagram for describing a display example of a virtualobject according to the present embodiment.

FIG. 11 is a diagram for describing a display example of a virtualobject according to the present embodiment.

FIG. 12 is a diagram for describing a display example of a virtualobject according to the present embodiment.

FIG. 13 is a diagram for describing a display example of a virtualobject according to the present embodiment.

FIG. 14 is a diagram for describing a display example of a virtualobject according to the present embodiment.

FIG. 15 is a flowchart illustrating an example of a flow ofpre-processing performed in a display device according to the presentembodiment.

FIG. 16 is a flowchart illustrating an example of a flow of a renderingprocess performed in a display device according to the presentembodiment.

FIG. 17 is a flowchart illustrating an example of a flow of a renderingprocess performed in a display device according to the presentembodiment.

FIG. 18 is a flowchart illustrating an example of a flow of a displaycontrol process according to an interaction performed in a displaydevice according to the present embodiment

FIG. 19 is a block diagram illustrating an example of a hardwareconfiguration of an information processing apparatus according to thepresent embodiment.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. In thisspecification and the appended drawings, structural elements that havesubstantially the same function and structure are denoted with the samereference numerals, and repeated explanation of these structuralelements is omitted.

In addition, there are cases in the present specification and thediagrams in which constituent elements having substantially the samefunctional configuration are distinguished from each other by affixingdifferent letters to the same reference numbers. For example, aplurality of constituent elements having substantially the samefunctional configuration are distinguished, like imaging units 10A and10B, if necessary. However, when there is no particular need todistinguish a plurality of constituent elements having substantially thesame functional configuration from each other, only the same referencenumber is affixed thereto. For example, when there is no particular needto distinguish imaging units 10A and 10B, they are referred to simply asimaging units 10.

The description will proceed in the following order.

1. Configuration example

1.1. External configuration example

1.2. Internal configuration example

2. Technical features

2.1. Overview of process

2.2. Variation of display process

2.3. Display region setting criteria

2.4. Display control based on interaction

2.5. Processing according to display region

2.6. Display control according to user

2.7. Other display control

3. Operation process example

4. Hardware configuration example

5. Conclusion

1. CONFIGURATION EXAMPLE 1.1. External Configuration Example

First, an example of an external configuration of an informationprocessing apparatus according to one embodiment of the presentdisclosure will be described with reference to FIG. 1.

FIG. 1 is a diagram illustrating an example of an external configurationof a display device 1 according to the present embodiment. The displaydevice 1 illustrated in FIG. 1 is a device also called smart glasses ora head mounted display (HMD). The display device 1 including a mountingunit 50 of a frame structure that surrounds, for example, the head ofthe user and is fixed to the head of the user through the mounting unit50. The display device 1 has a configuration in which a pair of displayunits 20A and 20B for the left eye and the right eye are arranged infront of the eyes of the user in the mounted state illustrated inFIG. 1. For example, a transmissive type display is used as the displayunit 20, and the display device 1 is able to enter a through state, thatis, a transparent or semi-transparent state, by controllingtransmittance of the transmissive type display. When the display unit 20is in the through state, there is no difficulty in normal life even ifthe user constantly wears the display device 1 as with glasses.

The display unit 20 is able to display a virtual object of the ARsuperimposed on the landscape of a real space by displaying images suchas text or drawings in the transparent or semi-transparent state. Inother words, the display device 1 may be implemented as a transmissivetype HMD. Note that, in the transmissive type HMD, preferably, thedisplay device 1 is fixed to the head of the user through the mountingunit 50, and a relative positional relation between the two eyes of theuser and the display unit 20 is fixed as illustrated in FIG. 1. This isbecause if the relative positional relation changes, a position on thedisplay for displaying the virtual object may change.

Further, the display unit 20 is also able to display the virtual objectsuperimposed on captured images of the real space captured by imagingunits 10A and 10B while displaying the captured images of the realspace. The display unit 20 is also able to display the virtual objectsuperimposed on the image of the virtual space while displaying imagessimilar to images obtained by imaging the virtual space through theimaging units 10A and 10B. In other words, the display device 1 may beimplemented as an immersive (video through type) HMD.

Alternatively, the display unit 20 may be implemented as an LED lightsource or the like that projects an image directly onto the retina ofthe user. In other words, the display device 1 may be implemented as aprojection type HMD.

Various types of content may be displayed on the display unit 20 as thevirtual object. For example, content may be data such as moving imagecontent including a movie or a video clip, still image content imaged bya digital still camera or the like, an electronic hook, or the like.Further, such content may all be data to be displayed such as computeruse data such as image data, text data, or spreadsheet data which isgenerated by the user using a personal computer or the like, a gameimage based on a game program, or the like

The imaging units 10A and 10B are arranged to perform imaging using areal space in a direction in which the user views as an imaging range ina state in which the user wears the display device 1. Each of theimaging units 10A and 10B may be implemented as a stereo camera capableof acquiring information indicating a distance in a direction in whichthe user views (hereinafter also referred to as “depth information”). Ina case in which each of the imaging units 10A and 10B is implemented asa stereo camera, the display device 1 is able to recognize a shape and aposture of the real object in the real space with a high degree ofaccuracy. Hereinafter, each of the imaging units 10A and 10B is alsoreferred to as an “outward facing stereo camera 10.”

On the other hand, imaging units 10C and 10D are arranged to performimaging using the direction of the user, more specifically, both eyes ofthe user, as the imaging range in the state in which the user wears thedisplay device 1. Each of the imaging units 10C and 10D may beimplemented as a stereo camera capable of acquiring the depthinformation in the direction of both eyes of the user. In a case inwhich each of the imaging units 10C and 10D is implemented as a stereocamera, the display device 1 is able to recognize an eyeball position, apupil position, a line-of-sight direction, or the like of the user witha high degree of accuracy. Hereinafter, each of the imaging units 10Cand 10D is also referred to as an “inward facing stereo camera 10.”

Further, although not illustrated in FIG. 1, the display device 1 mayhave a speaker or an earphone speaker. Further, the display device 1 mayinclude a microphone for acquiring an external sound.

The external appearance of the display device 1 illustrated in FIG. 1 isan example, and various structures in which the user wears the displaydevice 1 can be considered. The display device 1 may be constituted by amounting unit which is generally considered to be of an eyeglass type orhead mounted type, and at least in this embodiment, it is sufficientthat the display unit 20 is arranged close to the eyes of the user.Further, the display unit 20 may be configured such that a pair of unitscorresponding to both eyes are disposed or a single unit correspondingto one eye may be disposed.

Similarly, two speakers or two earphone speakers corresponding to theleft and right ears may be disposed, or one speaker or one earphonespeaker corresponding to one ear may be disposed. Further, one or moremicrophones may be disposed, and this arrangement is arbitrary.

The example of the external configuration of the display device 1according to the present embodiment has been described above. Next, anexample of an internal configuration of the display device 1 accordingto the present embodiment will be described with reference to FIG. 2.

Note that, in the following description, as an example, the displaydevice 1 is assumed to be implemented as a transmissive type HMD.Hereinafter, an image displayed on the display unit 20 (transmissiontype display) of the display device 1 (including a transparently visiblebackground and a superimposedly displayed virtual object) is alsoreferred to as a “real space image.”

1.2. Internal Configuration Example

FIG. 2 is a block diagram illustrating an example of an internalconfiguration of the display device 1 according to the presentembodiment. As illustrated in FIG. 2, the display device 1 includes animaging unit 10, the display unit 20, a storage unit 30, and a controlunit 40.

The imaging unit 10 has a function of imaging the real space andoutputting a captured image. For example, the imaging unit 10 is able tobe implemented as the outward facing stereo camera 10 and the inwardfacing stereo camera 10 illustrated in FIG. 1.

The display unit 20 has a function of displaying an image. For example,the display unit 20 may be implemented as a transmissive displayarranged at both eyes of the user illustrated in FIG. 1.

The storage unit 30 has a function of storing various information. Forexample, the storage unit 30 stores information related to the virtualobject displayed on the display unit 20. For example, the storage unit30 stores content such as an image or text serving as a source of thevirtual object.

The control unit 40 functions as an operation processing device and acontrol device and has a function of controlling an overall operation ofthe display device 1 according to various kinds of programs. The controlunit 40 is implemented by an electronic circuit such as a centralprocessing unit (CPU), a microprocessor, or the like. Note that thecontrol unit 40 may include a read only memory (ROM) that storesprograms, operation parameters, and the like to be used and a randomaccess memory (RAM) that temporarily stores parameters whichappropriately change and the like. Further, the control unit 40 mayinclude a graphics processing unit (GPU) and a video RAM (VRAM) forimage processing, and the like.

As illustrated in FIG. 2, the control unit 40 functions as a detectingunit 41, a storage control unit 43, and a display control unit 45, Thedetecting unit 41 has a function of detecting various kinds ofinformation from the captured image output from the imaging unit 10. Thestorage control unit 43 has a function of storing information in thestorage unit 30 and acquiring information stored in the storage unit 30.The display control unit 45 has a function of performing display controlfor causing images to be displayed on the display unit 20. Specifically,the display control unit 45 controls the display unit 20 such that thevirtual object is displayed superimposed on the real space (hereinafter,this is also described as “the display device 1 superimposedly displaysthe virtual object”),

The example of the internal configuration of the display device 1according to the present embodiment has been described above.

2. TECHNICAL FEATURES

Next, technical features of the display device 1 according to thepresent embodiment will be described in order.

2.1. Overview of Process

First, an overview of a process performed in the display device 1 willbe described with reference to FIG. 3. FIG. 3 is a diagram illustratingan overview of a process performed in the display device 1 according tothe present embodiment.

(1) Imaging Process

As illustrated in FIG. 3, the imaging unit 10 first performs an imagingprocess and acquires a captured image.

(2) Space Recognition Process

Next, as illustrated in FIG. 3, the detecting unit 41 performs a spacerecognition process on the basis of the captured image output from theimaging unit 10. Spatial information indicating the position and theposture of the display device 1 in the real space is acquired throughthe space recognition process.

For example, the spatial information may be an environment recognitionmatrix recognized by a publicly known image recognition technique suchas a structure from motion (SfM) technique or a simultaneouslocalization and mapping (SLAM) technique. For example, the environmentrecognition matrix indicates a relative position and a posture of acoordinate system of a reference environment (real space) relative to adevice-specific coordinate system of the display device 1. For example,in a case in which the SLAM method is used, a state variable including aposition, a posture, a velocity, and an angular velocity of a device anda position of at least one feature point included in a captured image isupdated for each frame of a captured image on the basis of a principleof an extended Kalman filter. Accordingly, it is possible to recognizethe position and posture of the reference environment based on theposition and posture of the device using an input image input from amonocular camera. Note that, for detailed description of the SLAMtechnique, see, for example, “Real-Time Simultaneous Localization andMapping with a Single Camera” (Andrew J. Davison, Proceedings of the 9thIEEE International Conference on Computer Vision Volume 2, 2003, pp.1403-1410).

In addition, the spatial information may be any information as long asit indicates the relative position and the posture of the imaging unitin the real space. For example, the environment recognition matrix maybe recognized on the basis of depth data obtained from a depth sensorthat may be installed in the imaging unit. The environment recognitionmatrix may also be recognized on the basis of output data output from anenvironment recognition system such as an infrared ranging system or amotion capture system. An example of such a technique is described in,for example, S. Izadi, et al, KinectFusion: Real-time 3D Reconstructionand Interaction Using a Moving Depth Camera, ACM Symposium on UserInterface Software and Technology, 2011, but the technique is notlimited thereto, and various publicly known techniques may be used forgeneration of the spatial information.

Alternatively, the spatial information may be generated by specifying arelative positional relation of each frame image through a stitchinganalysis for a series of frame images obtained by imaging the realspace. In this case, the stitching analysis may be two-dimensionalstitching analysis of attaching frame images onto a base plane orthree-dimensional stitching analysis of attaching frame images ontoarbitrary positions in a space.

Further, the spatial information may be acquired using an inertialsensor such as an acceleration sensor or a gyro sensor included in thedisplay device 1 together. In this case, the spatial information can beestimated at a higher speed. Further, it is possible to acquire thespatial information with a certain degree of accuracy even in a case inwhich it is difficult to acquire the spatial information on the basis ofthe captured image due to a motion blur or the like caused by fastmovement of the user (camera position).

(3) Real Object Recognition Process

Further, as illustrated in FIG. 3, the detecting unit 41 performs a realobject recognition process on the basis of the captured image outputfrom the imaging unit 10. For example, a real object included in thereal space image near the user who wears the display device 1 isdetected through the real object recognition process. Specifically, realobject information indicating a position and a shape of the real object,more accurately, a position and a shape of the real object in the realspace image (that is, unevenness of the real space viewed from theimaging unit 10), is acquired through the real object recognitionprocess.

For example, the real object information may be the depth information ofeach pixel and a degree of reliability of the depth information based onthe image obtained by the stereo camera. For example, the detecting unit41 acquires the real object information on the basis of a difference (abinocular parallax) in the real object in a plurality of captured imagesin which the same real space is an imaging target from differentviewpoints. Note that, due to a characteristic of stereo imagerecognition, the degree of reliability of the depth information relatedto a region in which a change in a color tone or the like is small maybe low. Further, the depth information may be acquired using anarbitrary technique such as a depth sensor of a time of flight (ToF)scheme.

Further, the real object information may be acquired by comparing afeature quantity calculated from the captured image output from theimaging unit 10 with a feature quantity of the real object stored in thestorage unit 30 in advance. In this case, the real object informationmay include identification information of a target real object andinformation indicating a position and a posture thereof. For example,the feature quantity may be calculated by a publicly known featurequantity calculation technique such as a SIFT technique or a randomferns technique.

In addition, the real object information may be acquired by recognizinga known diagram or symbol, an artificial marker (for example, a bar codeor a QR code (registered trademark)), or a natural marker.

(4) Gesture Recognition Process

Further, as illustrated in FIG. 3, the detecting unit 41 performs agesture recognition process on the basis of the captured image outputfrom the imaging unit 10. Gesture information indicating content of agesture performed by the user wearing the display device 1 or anotheruser nearby is acquired through the gesture recognition process. Thegesture information may be acquired using, for example, a techniquesimilar to that for the real object recognition process.

(5) Viewpoint Position Estimation Process

Next, as illustrated in FIG. 3, the display control unit 45 performs aviewpoint position estimation process. Viewpoint position informationindicating a position of the eye of the user wearing the display device1 is acquired through the viewpoint position estimation process. Forexample, the display control unit 45 estimates the viewpoint positioninformation on the basis of the spatial information acquired through thespace recognition process. The display control unit 45 may estimate theviewpoint position information further on the basis of the eyeballposition, the pupil position, the line-of-sight direction, or the likeof the user recognized from the image acquired by the inward facingstereo camera 10.

(6) Shielding Object Determination Process

Further, as illustrated in FIG. 3, the display control unit 45 performsa shielding object determination process. Shielding object informationindicating overlapping of the real objects viewed by the user wearingthe display device 1 is acquired through the shielding objectdetermination process. For example, the display control unit 45recognizes a positional relation between the real objects, a regionwhich is invisible to the user due to overlapping, and the like on thebasis of real object information.

(7) Display Region Setting Process

Further, as illustrated in FIG. 3, the display control unit 45 performsa process of setting a display region for the virtual object. A displayregion on which the virtual object is displayed (superimposed) is set inone region on the real space through the display region setting process.

For example, the display control unit 45 detects a plane region on thebasis of the real object information. The plane region refers to a flatsurface (having no or little unevenness) in the real object. Further,the display control unit 45 calculates, for example, an area, a normalline, a color, and a color variation degree of the plane region on thebasis of the depth information and the captured image. Then, the displaycontrol unit 45 sets a region suitable for displaying the virtual objectin the plane region as the display region. The display control unit 45may set a plane region calculated on the basis of the depth informationwith a higher degree of reliability as the display region preferentiallyover the plane region calculated on the basis of the depth informationwith a lower degree of reliability. The display region may have variousshapes such as a rectangle, a triangle, a polygon, or a circle. Forexample, the suitable region is a flat surface of the real object andindicates a surface having a sufficient area, and moreover, a smallangle between a vector from the imaging unit 10 to a plane and a normalline of a plane (that is, a surface facing the user). Further, asuitable region may be a region whose color change is flat. As describedabove, when the display region is set in the region suitable for thedisplay, collision between real world information and the virtual objectis prevented or reduced.

On the other hand, the display control unit 45 may set, for example, anarbitrary region (for example, a region in the air) on the space otherthan the surface of the real object as the display region. For example,in a case in which there is no flat surface in the real object, in acase in which there is no surface having a sufficient area, or in a easein which the angle between the vector from the imaging unit 10 to theplane and the normal line of the plane is large (that is, in a case inwhich it does not face the user, for example, a line-of-sight directionis substantially parallel to the plane), the display control unit 45sets a region in the air as the display region.

Further, the display control unit 45 may set the display region on thebasis of the gesture information. For example, the display control unit45 enlarges, reduces, or moves the display region according to theinstruction of the user.

The display control unit 45 lays out content in accordance with the setdisplay region and renders the laid-out content into the texture of thevirtual object.

(8) Virtual Object Rendering Process

Then, as illustrated in FIG. 3, the display control unit 45 performs avirtual object rendering process. For example, the display control unit45 decides a viewpoint for rendering on the basis of the viewpointposition information and the spatial information. Specifically, thedisplay control unit 45 performs projective transform of the displayregion set in the display region setting process into coordinates on thedisplay unit 20 on the basis of the viewpoint position information andthe spatial information. Hereinafter, the display region which hasundergone the projective transform is also referred to simply as a“display region.” Then, the display control unit 45 transforms thetexture of the virtual object through the GPU and writes resultinginformation in a VRAM corresponding to the display region which hasundergone the projective transform. Note that the display control unit45 may cause a shielded portion of the virtual object to disappear onthe basis of the shielding object information.

(9) Display Process

Finally, as illustrated in FIG. 3, the display unit 20 performs adisplay process. For example, the display unit 20 performs display basedon pixel information read from the VRAM through a display controller(for example, the display control unit 45). Accordingly, the virtualobject is displayed superimposed on the real space image.

2.2. Variation of Display Control Process

The display device 1 (for example, the display control unit 45)according to the present embodiment is able to simultaneously orselectively perform a plurality of types of superimposing processes. Forexample, the display device 1 is able to perform a first process ofdisplaying the virtual object superimposed on the real space at aposition which is based on the real object detected in the real space ora second process of displaying the virtual object superimposed on thereal space at a position which is not based on the real object detectedin the real space. Hereinafter, a variation of the display controlprocess performed by the display device will be described with referenceto FIG. 4.

FIG. 4 is a diagram for describing a display example of the virtualobject according to the present embodiment. As illustrated in FIG. 4,the user wearing the display device 1 is looking at a table. Virtualobjects 110A and 110B are displayed by the display device 1 and visibleto the user. For example, in the first process, the display device 1displays the virtual objects in accordance with a position, a posture, ashape, or the like of the real object. For example, in the exampleillustrated in FIG. 4, the virtual object 110A is displayed as if thevirtual object 110A stuck to a top surface of a table through the firstprocess. On the other hand, in the second process, the display device 1displays the virtual object regardless of the real object. For example,in the example illustrated in FIG. 4, the virtual object 110B isdisplayed as if the virtual object 110B floated while moving forwardalong the line of sight of the user through the second process.

Here, in this specification, a “position” may mean a position in thereal space perceived by the user (that is, real space coordinates) ormay mean a position on the display unit 20 (for example, a transmissivetype display) (that is, coordinates on a screen). However, the displayof the virtual object may be different depending on what it means. Forexample, even when there is a sense of perspective as the positionperceived by the user, it may be displayed at the same position on thedisplay unit 20.

The virtual object may be operation information which is a targetoperated by the user. For example, the user is able to inputinformation, and the like by touching the virtual object. In a case inwhich the virtual object floats in the air, it has been difficult forthe user to perform an operation while receiving feedback indicatingthat an operation is performed by touching the virtual object. On theother hand, in a case in which the virtual object is displayed on thesurface of the real object, the user is able to touch the virtual objectand touch the real object, and thus the user is able to perform anoperation while receiving feedback.

The first process and the second process will be described below indetail.

(1) First Process

(1a) Basic Policy

For example, the display device 1 may change the display of the virtualobject on the basis of the shape of the detected real object in thefirst process. Accordingly, the virtual object is more naturallysuperimposed on the real space when viewed by the user.

Specifically, the display device 1 changes the display of the virtualobject by changing the relative posture of the virtual object to thedetected real object and/or by transforming the shape of the virtualobject. For example, in a case in which the plane set as the displayregion in the detected real object does not faces the user, that is, ina case in which the display region is not orthogonal to theline-of-sight direction of the user, the display device 1 rotates thevirtual object by an angle formed by the display region and theline-of-sight direction. Further, for example, in a case in which theshape of the plane sot as the display region does not match the originalshape of the virtual object, the display device 1 transforms the shapeof the virtual object in accordance with the shape of the displayregion. Accordingly, the virtual object is viewed to be combined moreintegrally with the real object when viewed by the user. As a techniquefor changing the relative posture of the virtual object with respect tothe real object, for example, a publicly known technique such as atechnique described in JP 2012-221249A may be applied.

(1b) Superimposed Display on Surface of Real Object

The display device 1 may display the virtual object superimposed on thesurface of the detected real object. For example, the display device 1sets the surface of the real object as the display region, and maps thevirtual object to the surface of the real object, and displays thevirtual object. Accordingly, the virtual object looks to stick to thereal object when viewed by the user. In a case in which the virtualobject is displayed to be superimposed on the surface of the realobject, visibility is improved as compared with a case in which thevirtual object is displayed in the air which may overlap with thebackground. Note that the display device 1 may display the virtualobject superimposed on the same surface as the surface of the realobject (that is, on the surface of the real object) or may display thevirtual object superimposed on a surface separated from the surface (forexample, a plane region horizontally spaced apart from a plane region ofthe real object by several millimeters or several centimeters).

Here, the display device 1 may display the virtual object superimposedon a continuous surface among the surfaces of the detected real object.For example, the continuous surface is a region which has no shieldingobject or is not divided by a shielding object or the like in the planeregion. Further, for example, the continuous surface may be regarded asa surface which has a flat slope change and/or a flat color tone change.Accordingly, the virtual object is displayed to be more easily visiblewhen viewed by the user. Here, the present display example will bedescribed with reference to FIGS. 5 and 6.

FIG. 5 is a diagram for describing a display example of the virtualobject according to the present embodiment. In an example illustrated inFIG. 5, in a real space image 120, an envelope 122 is placed on thetable. For example, the display device 1 sets a surface of envelope 122which has a flat slope change and a flat color tone change as thedisplay region and displays a virtual object 124 on the envelope 122.

FIG. 6 is a diagram fir describing a display example of the virtualobject according to the present embodiment. In an example illustrated inFIG. 6, in a real space image 130, an envelope 132 is placed on thetable, and a cable 134 and a keyboard 136 are further placed on theenvelope 132. For example, the display device 1 sets a portion which isnot blocked by the cable 134 and the keyboard 136 serving as theshielding object in the surface of the envelope 132 which is thecontinuous surface having a flat slope change and a flat color tonechange as the display region, and causes a virtual object 138 to bedisplayed on the set region.

Further, the display device 1 may display the virtual objectsuperimposed on the surface determined to be horizontal or the surfacedetermined to be vertical among the surfaces of the detected realobject. The display device 1 may determine a horizontal or verticalsurface on the basis of the spatial information. For example, thedisplay device 1 displays the virtual object on a ground or a topsurface of a table or the like determined to be horizontal, or a walldetermined to be vertical, or the like. Accordingly, the virtual objectis displayed to be easily visible when viewed by the user.

Note that it is determined whether or not it is the around on the basisof an acceleration direction, a size of a plane, a relative position tothe display device 1, whether or not the plane is lowest among thehorizontal planes, or the like. In a case in which the ground is set asthe display region, the display device 1 is able to perform navigationaccording to a terrain.

(1c) Superimposed Display in Air

Further, the display device 1 may display the virtual objectsuperimposed on a position spaced apart from the detected real object.For example, the display device sets the display region in variousspaces such as a surface on an extension of the surface of the realobject, a space above the real object, or a space between the realobject and the user. More simply, the display device 1 displays thevirtual object as if it is floating in the air. Accordingly, it ispossible to display the virtual object in association with the realobject, for example, even when the surface of the real object has nosuitable display region. Note that, preferably, the display region setin the air faces the user.

(2) Second Process

For example, the display device 1 may display the virtual objectsuperimposed on the real space in the line-of-sight direction of theuser in the second process. For example, the display device 1 causes thevirtual object to move on with the movement of the line of sight of theuser on the basis of the spatial information and/or a result ofrecognizing an image captured by the inward facing stereo camera 10.Accordingly, the virtual object is seen to follow the line of sight at apredetermined distance in the air in front of the eye when viewed by theuser.

Further, the display device 1 may display the virtual objectsuperimposed on a position based on the head of the user in the secondprocess. For example, the display device 1 may cause the virtual objectto move on with the movement of the head of the user on the basis of thespatial information. Accordingly, for example, since the virtual objectis consistently within the field of view when viewed by the user, it ispossible to cause the eyes not to be blocked. For the similar purpose,the display device 1 may display the virtual object superimposed on aposition based on the display surface (for example, the display unit 20)in the second process.

Further, the display device 1 may display the virtual objectsuperimposed on an arbitrary position in the real space in the secondprocess. For example, the display device 1 displays the virtual objectsuperimposed on a position which is associated with neither the realobject nor the user. For example, the display device 1 may display thevirtual object at a position corresponding to a geographical position.

(3) Switching

The display device 1 may perform switching between the above-describedfirst process and second process. Accordingly, the display device 1 isable to select an appropriate process according to an environment of theuser and/or content. A switching criterion can be considered variously.

For example, the display device 1 may perform the switching on the basisof information related to the user and/or information related to thevirtual object. The information related to the user can be variouslyconsidered. Examples of the information related to the user may includeinformation related to the environment of the user, informationindicating an instruction given by the user, biometric information,position information, operation information, information related to anapplication being used, and information indicating a relation betweenthe user and the real object. Note that the information indicating therelation may indicate, for example, information indicating a positionalrelation, information indicating a posture relation, informationindicating a relation of attribute information, or the like. Theinformation related to the virtual object can be also variouslyconsidered. For example, the information related to the virtual objectmay include information indicating content or freshness of the virtualobject, a relation between the virtual object and the real object, orthe like. In addition to the above-mentioned information, for example,information indicating a relation between the user and the virtualobject may be considered as the information related to the user and theinformation related to the virtual object. The display device 1 mayperform the switching on the basis of at least one piece of theabove-mentioned information.

For example, the display device 1 may perform the switching inaccordance with the content of the virtual object. For example, thedisplay device 1 may display the virtual object at the position based onthe real object (for example, the surface of the real object) in a casein which a degree of urgency of content is low or may display thevirtual object at the position not based on the real object (forexample, the air in front of the eyes of the user) in a case in whichthe degree of urgency is high. Further, the display device 1 may displaythe virtual object at the position based on the real object (forexample, the surface of the real object or the air in the vicinity) in acase in which content is associated with the real object or may displaythe virtual object at the position not based on the real object (forexample, the air in front of the eyes of the user) in a case in whichcontent is not associated with the real object.

For example, the display device 1 may perform the switching inaccordance with a distance between the real object and the user. Forexample, the display device 1 may display the virtual object at theposition based on the real object in a case in which the real object inwhich the display region is settable is positioned nearby the user ormay display the virtual object at the position not based on the realobject in a case in which the real object in which the display region issettable is positioned far from the user.

For example, the display device 1 may switch in accordance with aninstruction given by the user.

2.3. Display Region Setting Criteria

The display device 1 (for example, the display control unit 45) is ableto set the display region on the basis of various criteria.

For example, the display device 1 may give priorities to the surface ofreal object, the space based on the real object, and the space not basedon the real object and set the display region

For example, the display device 1 may preferentially set the displayregion on the surface of the real object. Further, the display device 1may set the display region in the air in a case in which there is noappropriate display region on the surface of the real object or in acase in which it is requested by content. Further, the display region 1may set the display region facing the user in the air nearby the realobject for the plane region which has the flat color change but does notface the user. Further, the display device 1 may set the display regionin the air or set the display region on a plane that has already goneout of the field of view in a case in which a display region for displaya new virtual object is insufficient in a state in which a plurality ofvirtual object are displayed.

For example, the display device 1 may set the display region to avoidthe shielding object in a case in which the virtual object is displayedin the air. For example, the virtual object may be displayed in front ofthe real object although the display region is set at a position whichis originally hidden behind the real object and invisible. In such acase, for example, the display device 1 adjusts the display position toon the left or right side of the real object or set the display regionin front of the real object in order to avoid the real object.Accordingly, the parallax inconsistency is solved, and the burden on theeyes is reduced.

For example, the display device 1 may set the display region inaccordance with the degree of urgency of content. For example, forcontent with a high degree of urgency, the display device 1 sets thedisplay region by evaluating whether the content is located in themiddle of the field of view of the user rather than color flatness, orthe like. Further, the display device 1 may set the display region inthe air in front of the eyes to move on with the line of sight of theuser. Accordingly, the user is able to easily notice information havingthe high degree of urgency.

For example, the display device 1 may fix the display region to the realworld. For example, the display device 1 may store the position of thedisplay region and cause the virtual object to be displayed on the samedisplay region. Further, the display device 1 may fix the display regionfor each same or similar content. For example, the display device 1 maycause a virtual object of a video to be displayed on the wall surfaceand cause a virtual object of a text to be displayed on the table.

For example, the display device 1 may control the position at which thevirtual object is superimposedly displayed in accordance with thedistance between the real object and the user. For example, the displaydevice 1 preferentially sets the display region on the surface of thereal object located at a position close to the user. Accordingly, thevirtual object is displayed in a range in which the user is able toeasily perform an operation, for example, in a range in which the handreaches, and thus convenience is improved.

For example, the display device 1 may set the display region on thesurface of the real object associated with the content of the virtualobject or a region nearby the real object. For example, in a case inwhich a result of recognizing a real object placed on a table isdisplayed, the display device 1 may cause the recognition result to bedisplayed next to the real object.

For example, the display device 1 may set the display region on thebasis of information related to the real object. For example, thedisplay device 1 may set the display region to avoid the real objectwhich it is undesirable for the user to touch. For example, the displaydevice 1 does not set the display region on the surface of the realobject which is detected to be very hot by an infrared sensor.Accordingly, it is possible to increase safety of the user in a case inwhich the virtual object is an operation target of the user.

For example, the display device 1 may set a curved surface as thedisplay region in addition to a plane. In that case, the display device1 lays out the virtual object in a curved surface form and displays thevirtual object to stick to the curved surface. Further, in a case inwhich there is an undulating portion in the display region, the displaydevice 1 may lay out the virtual object in accordance with theundulating portion.

For example, in a case in which a plurality of virtual objects aredisplayed, the display device 1 may select the display region inaccordance with an area necessary for the virtual object. For example,the display device 1 causes a virtual object having a large display sizeto be displayed on a large plane and causes a virtual object having asmall display size to be displayed on a small plane with reference to adisplay size and the like described in content.

2.4. Display Control Based on Interaction

The display device 1 (for example, the display control unit 45) is ableto perform display control in accordance with an interaction with theuser.

For example, the display device 1 controls the position at which thevirtual object is superimposedly displayed in accordance with theinstruction given by the user. Resides, the display device 1 may switchcontent or control a display form (color or the like) in accordance withthe user instruction. Accordingly, a display more desirable to the useris performed, and a comfortable operation for the virtual object isimplemented. For example, the instruction may be given by the user by agesture performed by a hand or the like, the line of sight, markerembedding, or the like. Here, the present display example will bedescribed with reference to FIGS. 7 to 10.

FIG. 7 is a diagram for describing a display example of the virtualobject according to the present embodiment. In an example illustrated inFIG. 7, the display device 1 sets the display region in accordance withthe gesture. As illustrated in FIG. 7, if the user drags a virtualobject 140 being displayed in the air onto a plane 142 with a finger,the display device 1 moves the virtual object 140 to the surface of theplane 142 and displays the virtual object 140.

FIG. 8 is a diagram for describing a display example of the virtualobject according to the present embodiment. An example illustrated inFIG. 8 is an example in which the display device 1 sets the displayregion in accordance with the line of sight. An illustrated in FIG. 8,in a case in which the user looks at the plane 142 for a predeterminedtime, the display device 1 moves the virtual object 140 being displayedin the air to the surface of the plane 142 and displays the virtualobject 140.

FIG. 9 is a diagram for describing a display example of the virtualobject according to the present embodiment. In an example illustrated inFIG. 9, the display device 1 sets the display region in accordance withthe marker. As illustrated in FIG. 9, in a case in which a plane 152 inwhich a specific marker 150 such as a QR code (a registered trademark)is embedded is detected, the display device 1 moves a virtual object 154corresponding to the marker 150 being displayed in the air to the plane152 and displays the virtual object 154.

FIG. 10 is a diagram for describing a display example of the virtualobject according to the present embodiment. In an example illustrated inFIG. 10, the display device 1 sets the display region in accordance withthe marker. As illustrated in FIG. 10, a marker 162 is embedded in atransparent panel 164 such as a glass panel. For example, the displaydevice 1 may irradiating the panel 164 with a specific wavelength anddetect the marker 162. Accordingly, in a real space image 160, the panel164 that is seen to be just a transparent plane with the naked eye isused as the display region for a virtual object 166.

Note that, in the examples illustrated in FIGS. 7 to 10, the virtualobject is moved to the plane, but the present technology is not limitedto these examples. For example, the display device 1 may move thevirtual object from a plane to a plane in accordance with the userinstruction, move the virtual object from the air to the air, or movethe virtual object from a plane to the air.

Further, in a case in which the user puts a graspable plane in front ofthe eyes or creates a plane with the palm of the hand, the displaydevice 1 is also able to set the display region on these planes. As amethod of detecting the user operation on the virtual object beingdisplayed on such a plane, the display device 1 may acquire vibrationinformation from a wearable device such as a watch type worn on a bodyof the user and use the vibration information for detection of the useroperation. In this case, the display device 1 is able to accuratelyidentify whether or not the finger of the user touches the plane throughthe vibration information, and thus the operability of the touchoperation is further improved. Furthermore, since the display device 1is able to distinguish the touch operation of touching the plane and thegesture operation performed in front of the plane, the operationsperformed using the operations separately or together can be implementedwith a higher degree of accuracy.

2.5. Processing According to Display Region

The display device 1 (for example, the display control unit 45) is ableto perform various kinds of processing according to the display region.

For example, the display device 1 may process the virtual object inaccordance with the shape of the display region. Specifically, thedisplay device 1 controls the layout of the virtual object in accordancewith the shape of display region. The display device 1 may employ apublicly known technique such as an HTML renderer as a layout engine.For example, a technique of switching a screen layout in accordance witha window size, a type of device, or the like or a technique of changingthe layout of texts or the like in accordance with an arrangement ofimages or the like in the HTML may be adopted. Here, the present displayexample will be described with reference to FIG. 11.

FIG. 11 is a diagram for describing a display example of the virtualobject according to the present embodiment. In an example illustrated inFIG. 11, in a real space image 170, a virtual object 174 is displayed ona display region excluding a real object 172. As illustrated in FIG. 11,texts included in the virtual object 174 start a new line in accordancewith the shape of the display region.

However, the shape of the display region may change. For example, a newreal object (shielding object) may be added to the display region, or anexisting real object may be deleted or moved. In such cases, the displaydevice 1 may control the layout of the virtual object in accordance withthe change in the shape of the display region. Here, the present displayexample will be described with reference to FIG. 12.

FIG. 12 is a diagram for describing a display example of the virtualobject according to the present embodiment. The display exampleillustrated in FIG. 12 illustrates a change from the display exampleillustrated in FIG. 11. As illustrated in FIG. 12, in a real space image180, a real object 182 is moved to a region which is included in thedisplay region in the real space image 170 illustrated in FIG. 11, andthe layout of a virtual object 184 is changed in accordance with anarrowed shape, and a position at which texts start a new line ischanged. Further, in a case in which a new plane is set as the displayregion, it is desirable that a new display region be set on a planewhich is close in a position, an attribute of a real object on which itis superimposed, a slope, or the like to a previous display region.

Further, the display device 1 may process the content of the virtualobject in accordance with the display region. For example, the displaydevice 1 switches the content to be displayed in accordance with ashape, a color, a size, or the like of the display region. Furthermore,the display device 1 may switch the content to be displayed on the basisof a real object serving as a reference for the position of the displayregion, a real object close to the display region, or the like. Here,the present display example will be described with reference to FIG. 13.

FIG. 13 is a diagram for describing a display example of the virtualobject according to the present embodiment. In the example illustratedin FIG. 13, in a real space image 190, a virtual object 192 of an analogclock is displayed on a wall surface. For example, this arrangement isperformed because the display region is the wall surface, and the areaof the display region is large. On the other hand, in a real space image194, a virtual object 196 of a digital clock is displayed on a table.For example, this arrangement is performed because the display region isthe surface of the table, and the area of the display region is small.

Further, the display device 1 may set the display region or process thevirtual object in accordance with the background.

For example, the display device 1 may control the position at which thevirtual object is superimposedly displayed in accordance with thebackground of the region on which the virtual object can besuperimposedly displayed and a display form of the virtual object. Forexample, it may be hard to see content of the virtual object dependingon brightness of the background. Further, it may be hard to see contentof the virtual object depending on a relation between the color tone ofthe background and the color tone of the virtual object. Inconsideration of such a case, the display device 1 may set the displayregion to avoid the region in which it is hard to view the virtualobject. Accordingly, the virtual object is displayed to be more easilyvisible when viewed by the user. Here, the present display example willbe described with reference to FIG. 14.

FIG. 14 is a diagram for describing a display example of the virtualobject according to the present embodiment. In an example illustrated inFIG. 14, in a real space image 200, respective virtual objects aredisplayed on display regions 202, 204, and 206 floating in the air. Forexample, in the display region 202, there is an electric lamp in thebackground, and it is hard to view content of the virtual object due toinfluence of brightness. Further, in the display region 204, it is hardto view content of the virtual object because there is a real object inthe background. On the other hand, in the display region 206, thebackground is flat, and thus content of the virtual object is clearlyseen. Therefore, the display device 1 preferentially employs the displayregion 206.

Furthermore, the display device 1 may control the display form of thevirtual object in accordance with the background of the region on whichthe virtual object is superimposedly displayed. For example, it may behard to see content of the virtual object depending on a relationbetween the color tone of the background of the display region and thecolor tone of the virtual object. In consideration of such a case, forexample, the display device 1 performs processing of emphasizing thevirtual image using a color complementary to the color of the backgroundor not emphasizing the virtual image using a color close to the color ofthe background. This processing function may be included in contentitself

Further, the display device 1 may perform display control correspondingto a real object in the vicinity of the display region. For example, thedisplay device 1 may cause a virtual object including informationrelated to a sentence to be displayed next to a notebook on a table in acase in which the user is writing the sentence on the notebook on thetable. The user is able to execute, for example, information search byoperating the virtual object. Further, the display device 1 may updatethe display region in real time, for example, by causing the virtualobject to be displayed on a new empty region made on the table as aresult of moving in a ease in which the notebook is moved.

2.6. Display Control in Accordance with the User

The display device 1 (for example, display control unit 45) is able toperform display control according to the user.

For example, the display device 1 may perform display control accordingto an action of the user. For example, the display device 1 excludes aregion blocking the center of the field of view from a display regionsetting target while the user is driving a vehicle. Further, the displaydevice 1 may perform display control according to whether or not theuser is stationary. For example, in a case in which the user isstationary, for example, in a case in which the user faces a table, afixed display region is set on the table, and in a case in which theuser is moving, for example, in a case in which the user is running, adisplay region moving together with the user is set. In other words, thedisplay device 1 fixedly sets the display region in the coordinatesystem of the real world while the user is stationary, and fixedly setsthe display region in a coordinate system in which the user serves as anorigin while the user is moving.

For example, the display device 1 may perform display control accordingto a position of the user. For example, in a ease in which mapinformation is displayed, the display device 1 causes the mapinformation to be displayed on the ground when the user is far from thedestination, and causes a virtual object indicating the destination tobe displayed nearby the destination when the user is close to thedestination.

For example, the display device 1 may perform display control accordingto the line of sight of the user. For example, in a case in which thedisplay region disappears fro the field of view, the display device 1may set a new display region while storing the position of the displayregion, display the virtual object, and causes the virtual object toreturn to the previous display region when the previous region comesinto the field of view again. Further, the display device 1 may displaya virtual object in which the line of sight does not concentrate in anormal mode and display a virtual object in which the line of sightconcentrates in a notable mode in which a noticeable color, layout, orthe like which is easily visible is used.

2.7. Other Display Control

The display device 1 (for example, the display control unit 45) is ableto perform various other display controls.

For example, the display device 1 may perform display control accordingto a time. For example, the display device 1 displays a virtual objectindicating news, or a weather forecast, or the like in the morning anddisplays a virtual object indicating a recipe site in the evening.

For example, the display device 1 may deal a plurality of regions as onedisplay region. For example, in a case in which a cable is placed on atable, and a plane is divided into two, the display device 1 may setboth of the two divided planes as the display region. In that case, thedisplay device 1 simply sets a region where the cable is simply placedas a region in which rendering is prevented, and layouts content toavoid the region.

For example, in a case in which the number of virtual objects is largeor in a case in which the color of the virtual object is familiar withthe surroundings, it may be difficult to see the virtual object whenviewed by the user. In such a case, the display device 1 may temporarilychange color or perform vibration, for example, in a case in which a newvirtual object appears or in a case in which there is a change in avirtual object. Accordingly, the user is able to notice a new or changedvirtual object.

The technical features of the display device 1 according to the presentembodiment have been described above. Next, an operation process exampleof the display device 1 according to the present embodiment will bedescribed with reference to FIGS. 15 to 18.

3. OPERATION PROCESS EXAMPLE

(1) Pre-Processing

FIG. 15 is a flowchart illustrating an example of the flow ofpre-processing executed in the display device 1 according to the presentembodiment. The present flow corresponds to the real object recognitionprocess, the shielding object determination process, and the displayregion setting process described above with reference to FIG. 3.

As illustrated in FIG. 15, the detecting unit 41 acquires the depthinformation of each pixel and the degree of reliability of the depthinformation from the image obtained by the outward stereo camera 10(step S102). Then, the display control unit 45 detects the plane regionon the surface of the real object on the basis of the depth informationand the degree of reliability (step S104). Then, the display controlunit 45 calculates the feature of each plane region (step S106). Thefeatures of each plane region include, for example, an area, a normalline, a color, and a color variation degree of the plane region and maybe calculated on the basis of, for example, the depth information andthe degree of reliability.

Then, the display control unit 45 determines whether or not there is aplane region suitable for display (step S108). For example, the displaycontrol unit 45 determines that there is a plane region suitable fordisplay in a case in which there is a plane region which has a flatcolor change, faces the user, and is small in a distance from the user,and has no virtual object displayed thereon. Further, the displaycontrol unit 45 determines that there is no plane region suitable fordisplay in a case in which there is no plane region satisfying theconditions.

When it is determined that there is a plane region suitable for display(YES in step S108), the display control unit 45 sets the display regionin the plane region (step S110), On the other hand, in a ease in whichit is determined that there is no plane region suitable for display (NOin step S108), the display control unit 45 sets the display region inthe air (step S112).

The display control unit 45 then creates the texture of the virtualobject by laying out and rendering content in accordance with the setdisplay region (step S114). Then, the process according to the presentflow ends.

(2) Rendering Process

FIG. 16 is a flowchart illustrating an example of the flow of therendering process performed in the display device 1 according to thepresent embodiment. The present flow corresponds to the viewpointposition estimation process, the virtual object rendering process, andthe display process described above with reference to FIG. 3 and isexecuted after the pre-processing described above with reference to FIG.15.

As illustrated in FIG. 16, first, the detecting unit 41 acquires thespatial information and the viewpoint position information (step S202).Then, the display control unit 45 performs projective transform of thedisplay region set in pre-processing into the coordinates on the displayunit 20 on the basis of the spatial information and the viewpointposition information (step S204). Then, the display control unit 45 mapsthe texture of the virtual object generated in the pre-processing to thedisplay region which has undergone the projective transform and displaysthe virtual object (step S206).

The display device 1 performs the above-described process for eachframe, in a ease in which the rendering process does not end (NO in stepS208), the process is performed again from step S202 for the next frame.In a case in which the rendering process ends (YES in step S208), theprocess according to the present flow ends.

FIG. 17 is a flowchart illustrating an example of the flow of therendering process performed in the display device 1 according to thepresent embodiment. The present flow is a modification of the flowillustrated in FIG. 16.

As illustrated in FIG. 17, first in steps S302 to S306, a processsimilar to steps S202 to S206 described above with reference to FIG. 16is performed. Then, in steps S308 to S312, a process similar to stepsS102 to S106 described above with reference to FIG. 15 is performed.

Then, the display control unit 45 determines whether or not the displayregion is changed (step S314). For example, the display control unit 45determines whether or not the display region is changed to a moresuitable region (plane region or air) in accordance with movement of thefield of view, addition/deletion/movement of the real object, a changein the degree of urgency of content, the presence or absence of the userinstruction, or the like.

In a case in which the display region is determined to be changed (YESin step S314), the display control unit 45 resets the display region toa new region (step S316). On the other hand, in a case in which thedisplay region is determined not to be changed (NO in step S314), thesetting of the display region is maintained.

The display device 1 performs the above-described process for eachframe. In a case in which the rendering process does not end (NO in stepS318), the process is performed again from step S302 for the next frame.In a case in which the rendering process ends (YES in step S318), theprocess according to the present flow ends.

(3) Display Control Process According to Interaction

FIG. 18 is a flowchart illustrating an example of the flow of a displaycontrol process according to the interaction performed in the displaydevice 1 according to the present embodiment.

As illustrated in FIG. 18, first, the detecting unit 41 acquires a userinstruction related to the setting of the display region (step S402).For example, the detecting unit 41 detects the gesture, the line ofsight, or the plane in which a marker is embedded.

Then, the display control unit 45 sets the display region on the basisof the user instruction (step S404). For example, the display controlunit 45 sets the display region on a plane to which the user drags thevirtual object with the finger. Further, the display control unit 45sets the display region on a plane at which the user looks for apredetermined time. Further, the display control unit 45 sets thedisplay region on a plane in which a specific marker is embedded.

Then, the display control unit 45 causes the virtual object to bedisplayed on the set display region (step S406).

Then, the detecting unit 41 acquires the user instruction on the virtualobject (step S408). For example, the detecting unit 41 recognizes atouch on the plane or the air in which the display region is set on thebasis of the depth information or the like. Further, the detecting unit41 determines whether or not the user is wearing a wearable device (stepS410), and in a case in which the user is wearing a wearable device, thedetecting unit 41 acquires vibration information from the wearabledevice (step S412). Then, the display control unit 45 displays afeedback on the acquired user instruction (step S414). In a case inwhich the vibration information is acquired, the display control unit 45displays feedback further on the basis of the vibration information.Then, the process according to the present flow ends.

4. HARDWARE CONFIGURATION EXAMPLE

Finally, a hardware configuration of an information processing apparatusaccording to the present embodiment will be described with reference toFIG. 19, FIG. 19 is a block diagram illustrating an example of thehardware configuration of the information processing apparatus accordingto the present embodiment. Meanwhile, the information processingapparatus 900 illustrated in FIG. 19 may realize the display device 1illustrated in FIG. 2, for example. Information processing by thedisplay device 1 according to the present embodiment is realizedaccording to cooperation between software and hardware described below.

As illustrated in FIG. 19, the information processing apparatus 900includes a central processing unit (CPU) 901, a read only memory (ROM)902, a random access memory (RAM) 903 and a host bus 904 a. In addition,the information processing apparatus 900 includes a bridge 904, anexternal bus 904 b, an interface 905, an input device 906, an outputdevice 907, a storage device 908, a drive 909, a connection port 911 anda communication device 913. The information processing apparatus 900 mayinclude a processing circuit such as a DSP or an ASIC instead of the CPU90 i or along therewith.

The CPU 901 functions as an arithmetic processing device and a controldevice and controls the overall operation in the information processingapparatus 900 according to various programs. Further, the CPU 901 may bea microprocessor. The ROM 902 stores programs, operation parameters andthe like used by the CPU 901. The RAM 903 temporarily stores programsused in execution of the CPU 901, parameters appropriately changed inthe execution, and the like. For example, the CPU 901 is able toconstitute the detecting unit 41, the storage control unit 43, and thedisplay control unit 45 illustrated in FIG. 2.

The CPU 901, the ROM 902 and the RAM 903 are connected by the host bus904 a including a CPU bus and the like. The host bus 904 a is connectedwith the external bus 904 b such as a peripheral componentinterconnect/interface (PCI) bus via the bridge 904. Further, the hostbus 904 a, the bridge 904 and the external bus 904 b are not necessarilyseparately configured and such functions may be mounted in a single bus.

The input device 906 is realized by a device through which a user inputsinformation, such as a mouse, a keyboard, a touch panel, a button, amicrophone, a switch, and a lever. In addition, the input device 906 maybe a remote control device using infrared ray or other electric waves orexternal connection equipment such as a cellular phone or a PDAcorresponding to operation of the information processing apparatus 900,for example. Furthermore, the input device 906 may include an inputcontrol circuit or the like which generates an input signal on the basisof information input by the user using the aforementioned input meansand outputs the input signal to the CPU 901, for example. The user ofthe information processing apparatus 900 may input various types of dataor order a processing operation for the information processing apparatus900 by operating the input device 906.

The input device 906 may be realized as an imaging device, for example.The imaging device includes a lens system composed of an imaging lens,an iris, a zoom lens, a focus lens and the like, a driving system forcausing the lens system to perform a focus operation and a zoomoperation, a solid-state imaging sensor array for photo-electricallyconverting imaging light acquired through the lens system to generate animaging signal, and the like. The solid-state imaging sensor array maybe realized by a charge coupled device (CCD) sensor array or acomplementary metal oxide semiconductor (CMOS) sensor array, forexample. The imaging device outputs data of shot images in the form ofdigital signals. The imaging device may form the imaging unit 10illustrated in FIGS. 1 and 2, for example.

The output device 907 is formed by a device that may visually or aurallynotify the user of acquired information. As such devices, there is adisplay device such as a CRT display device, a liquid crystal displaydevice, a plasma display device, an EL display device, a laserprojector, an LED projector or a lamp, a sound output device such as aspeaker and a headphone, a printer device and the like. The outputdevice 907 outputs results acquired through various processes performedby the information processing apparatus 900, for example. Specifically,the display device visually displays results acquired through variousprocesses performed by the information processing apparatus 900 invarious forms such as text, images, tables and graphs. On the otherhand, the sound output device converts audio signals composed ofreproduced sound data, audio data and the like into analog signals andaurally outputs the analog signals. The aforementioned display devicemay form the display unit 20 illustrated in FIGS. 1 and 2, for example.

The storage device 908 is a device for data storage, formed as anexample of a storage unit of the information processing apparatus 900.For example, the storage device 908 is realized by a magnetic storagedevice such as an HDD, a semiconductor storage device, an opticalstorage device, a magneto-optical storage device or the like. Thestorage device 908 may include a storage medium, a recording device forrecording data on the storage medium, a reading device for reading datafrom the storage medium, a deletion device for deleting data recorded onthe storage medium and the like. The storage device 908 stores programsand various types of data executed by the CPU 901, various types of dataacquired from the outside and the like. The storage device 908 may formthe storage unit 30 illustrated in FIG. 2, for example.

The drive 909 is a reader/writer for storage media and is included in orexternally attached to the information processing apparatus 900. Thedrive 909 reads information recorded on a removable storage medium suchas a magnetic disc, an optical disc, a magneto-optical disc or asemiconductor memory mounted thereon and outputs the information to theRAM 903. In addition, the drive 909 can write information on theremovable storage medium.

The connection port 911 is an interface connected with externalequipment and is a connector to the external equipment through whichdata may be transmitted through a universal serial bus (USB) and thelike, for example.

The communication device 913 is a communication interface formed by acommunication device for connection to a network 920 or the like, forexample. The communication device 913 is a communication card or thelike for a wired or wireless local area network (LAN), long termevolution (LTE), Bluetooth (registered trademark) or wireless USB(WUSB), for example. In addition, the communication device 913 may be arouter for optical communication, a router for asymmetric digitalsubscriber line (ADSL), various communication modems or the like. Forexample, the communication device 913 may transmit/receive signals andthe like to/from the Internet and other communication apparatusesaccording to a predetermined protocol, for example, TCP/IP or the like.

Further, the network 920 is a wired or wireless transmission path ofinformation transmitted from devices connected to the network 920. Forexample, the network 920 may include a public circuit network such asthe Internet, a telephone circuit network or a satellite communicationnetwork, various local area networks (LANs) including Ethernet(registered trademark), a wide area network (WAN) and the like. Inaddition, the network 920 may include a dedicated circuit network suchas an internet protocol-virtual private network (IP-VPN).

Hereinbefore, an example of a hardware configuration capable ofrealizing the functions of the information processing apparatus 900according to this embodiment is shown. The respective components may beimplemented using universal members, or may be implemented by hardwarespecific to the functions of the respective components. Accordingly,according to a technical level at the time when the embodiments areexecuted, it is possible to appropriately change hardware configurationsto be used.

In addition, a computer program for realizing each of the functions ofthe information processing apparatus 900 according to the presentembodiment as described above may be created, and may be mounted in a PCor the like. Furthermore, a computer-readable recording medium on whichsuch a computer program is stored may be provided. The recording mediumis a magnetic disc, an optical disc, a magneto-optical disc, a flashmemory, or the like, for example. Further, the computer program may bedelivered through a network, for example, without using the recordingmedium.

5. CONCLUSION

One embodiment of the present disclosure has been described above indetail with reference to FIGS. 1 to 19. As described above, the displaydevice 1 according to the present embodiment performs the first processof displaying the virtual object superimposed on the real space at aposition which is based on the real object or the second process ofdisplaying the virtual object superimposed on the real space at aposition which is not based on the real object. Accordingly, the methodof implementing the virtual objects using the AI technology isdiversified.

For example, in the first process, the display of the virtual object maybe changed in accordance with the shape of the detected real object.More specifically, the virtual object is displayed on the surface of thereal object in accordance with the shape of the real object. Throughsuch a display, the virtual object becomes more familiar to the realworld, and the user is able to accept the mixture of the virtual objectand the real object with no discomfort feeling. Further, the displaydevice 1 is also able to cause newly arriving information of a clock,news, an e-mail, or an SNS or the like to be displayed on a vacantregion of a wall or the like in accordance with a degree of importancein an emphasized manner regardless of a place.

Accordingly, the user is able to easily acquire information withoutchecking a terminal such as a smartphone individually. Further, thedisplay device 1 is able to display the virtual object in accordancewith the shape of the vacant region even in a case in which furniture islined up in front of a wall, or objects are scattered on a table.

Further, in the second process, the virtual object may be superimposedlydisplayed on the position based on the head of the user. Accordingly,for example, although the virtual object consistently comes into thefield of view when viewed by the user, the state in which the eyes arenot blocked can be maintained.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

For example, in the above embodiment, the example in which the displaydevice 1 is formed as a single device has been described, but thepresent technology is not limited to this example. For example, part orall of the display device 1 may be constituted as separate devices. Forexample, in the functional configuration example of the display device 1illustrated in FIG. 2, the storage unit 30 and the control unit 40 maybe installed in a device such as a server on a cloud connected with thedisplay device including the imaging unit 10 and the display unit 20 viaa network or the like.

Note that it is not necessary for the processing described in thisspecification with reference to the flowchart and the sequence diagramto be executed in the order shown in the flowchart. Some processingsteps may be performed in parallel. Further, some of additional stepscan be adopted, or some processing steps can be omitted.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art from the description of this specification.

Additionally, the present technology may also be configured as below.

(1)

An information processing apparatus including:

a display control unit configured to perform a first process ofdisplaying a virtual object superimposed on a real space at a positionwhich is based on a real object detected in the real space or a secondprocess of displaying the virtual object superimposed on the real spaceat a position which is not based on the real object detected in the realspace.

(2)

The information processing apparatus according to (1),

in which the display control unit performs switching between the firstprocess and the second process.

(3)

The information processing apparatus according to (2),

in which the display control unit performs the switching on the basis ofinformation related to a user and/or information related to the virtualobject,

(4)

The information processing apparatus according to (3),

in which the display control unit performs the switching in accordancewith content of the virtual object.)

(5) The information processing apparatus according to (3) or (4),

in which the display control unit performs the switching in accordancewith a distance between the real object and the user.

(6)

The information processing apparatus according to (1),

in which the display control unit changes the display of the virtualobject on the basis of a shape of the detected real object in the firstprocess.

(7)

The information processing apparatus according to (6),

in which the display control unit changes the display of the virtualobject by changing a relative posture of the virtual object with respectto the detected real object and/or by transforming the shape of thevirtual object.

(8)

The information processing apparatus according to (6) or (7),

in which the display control unit displays the virtual objectsuperimposed on a surface of the detected real object.

(9)

The information processing apparatus according to (8), in which thedisplay control unit displays the virtual object superimposed on acontinuous surface among surfaces of the detected real object.

(10)

The information processing apparatus according to (8) or (9), in whichthe display control unit displays the virtual object superimposed on asurface determined to be horizontal or a surface determined to bevertical among surfaces of the detected real object.

(11)

The information processing apparatus according to any one of (8) to(10),

in which the display control unit displays the virtual objectsuperimposed on a position spaced apart from the detected real object.

(12)

The information processing apparatus according to any one of (1) to(11),

in which the display control unit displays the virtual objectsuperimposed on a position based on a head of a user or a real space ina line-of-sight direction of the user in the second process.

(13)

The information processing apparatus according to any one of (1) to(11),

in which the display control unit displays the virtual objectsuperimposed on an arbitrary position in the real space in the secondprocess.

(14)

The information processing apparatus according to any one of (1) to(13),

in which the display control unit controls a position on which thevirtual object is superimposedly displayed in accordance with a distancebetween the real object and a user.

(15)

The information processing apparatus according to any one of (1) to(14),

in which the display control unit controls a position on which thevirtual object is superimposedly displayed in accordance with aninstruction given by a user.

(16)

The information processing apparatus according to any one of (1) to(15),

in which the display control unit controls a position on which thevirtual object is superimposedly displayed in accordance with abackground of a region on which the virtual object is superimposedlydisplayable and a display form of the virtual object.

(17)

The information processing apparatus according to any one of (1) to(16),

in which the display control unit controls a display form of the virtualobject in accordance with a background of a region on which the virtualobject is superimposedly displayed.

(18)

The information processing apparatus according to any one of (1) to(17),

in which the virtual object is a target operated by a user.

(19)

An information processing method including:

performing, by a processor, a first process of displaying a virtualobject superimposed on a real space at a position which is based on areal object detected in the real space or a second process of displayingthe virtual object superimposed on the real space at a position which isnot based on the real object detected in the real space.

(20)

A program causing a computer to function as:

a display control unit configured to perform a first process ofdisplaying a virtual object superimposed on a real space at a positionwhich is based on a real object detected in the real space or a secondprocess of displaying the virtual object superimposed on the real spaceat a position which is not based on the real object detected in the realspace.

REFERENCE SIGNS LIST

-   1 display device-   10 imaging unit-   20 display unit-   30 storage unit-   40 control unit-   41 detecting unit-   43 storage control unit-   45 display control unit-   50 mounting unit

The invention claimed is:
 1. An information processing apparatuscomprising: a display control unit configured to determine whether areal object having a flat surface with a suitable region to display avirtual object is positioned within a range in accordance with adistance between the real object and a mobile terminal, perform, when itis determined that the real object having the flat surface with thesuitable region is positioned within the range, a first process ofdisplaying the virtual object, wherein a display of the virtual objectis changed and the virtual object is superimposed on a display regionwithin the suitable region of the flat surface of the real object basedon a user operation in the first process, and perform, when it isdetermined that the real object does not have the flat surface with thesuitable region positioned within the range, a second process ofdisplaying the virtual object superimposed in midair in front of themobile terminal based on the user operation in the second process,wherein the display control unit determines that the real object has theflat surface with the suitable region to display the virtual objectbased on an area of the suitable region being greater than an areathreshold and an angle formed between a normal line of the suitableregion and a line-of-sight direction being less than an angularthreshold, and wherein the display control unit is implemented via atleast one processor.
 2. The information processing apparatus accordingto claim 1, wherein the user operation includes a drag operation.
 3. Theinformation processing apparatus according to claim 1, wherein, in thefirst process, a shape of the virtual object is transformed.
 4. Theinformation processing apparatus according to claim 3, wherein, in thefirst process, the virtual object is rotated by the angle formed betweenthe normal line of the suitable region of the flat surface and theline-of-sight direction.
 5. The information processing apparatusaccording to claim 3, wherein the shape of the virtual object istransformed in accordance with a shape of the flat surface.
 6. Theinformation processing apparatus according to claim 1, wherein the flatsurface is a horizontal surface.
 7. The information processing apparatusaccording to claim 1, wherein, in the first process, the virtual objectis superimposed in a same plane region as the flat surface.
 8. Theinformation processing apparatus according to claim 1, wherein, in thefirst process, the virtual object is superimposed in a plane region thatis separated from a plane region of the flat surface.
 9. The informationprocessing apparatus according to claim 1, wherein the display controlunit further determines that the real object has the flat surface withthe suitable region to display the virtual object based on a colorchange of the suitable region being less than a color change threshold.10. An information processing method comprising: determining whether areal object having a flat surface with a suitable region to display avirtual object is positioned within a range in accordance with adistance between the real object and a mobile terminal; performing, whenit is determined that the real object having the flat surface with thesuitable region is positioned within the range, a first process ofdisplaying the virtual object, wherein a display of the virtual objectis changed and the virtual object is superimposed on a display regionwithin the suitable region of the flat surface of the real object basedon a user operation in the first process; and performing, when it isdetermined that the real object does not have the flat surface with thesuitable region positioned within the range, a second process ofdisplaying the virtual object superimposed in midair in front of themobile terminal based on the user operation in the second process,wherein the real object is determined to have the flat surface with thesuitable region to display the virtual object based on an area of thesuitable region being greater than an area threshold and an angle formedbetween a normal line of the suitable region and a line-of-sightdirection being less than an angular threshold.
 11. A non-transitorycomputer-readable storage medium having embodied thereon a program,which when executed by a computer causes the computer to execute amethod, the method comprising: determining whether a real object havinga flat surface with a suitable region to display a virtual object ispositioned within a range in accordance with a distance between the realobject and a mobile terminal; performing, when it is determined that thereal object having the flat surface with the suitable region ispositioned within the range, a first process of displaying the virtualobject, wherein a display of the virtual object is changed and thevirtual object is superimposed on a display region within the suitableregion of the flat surface of the real object based on a user operationin the first process; and performing, when it is determined that thereal object does not have the flat surface with the suitable regionpositioned within the range, a second process of displaying the virtualobject superimposed in midair in front of the mobile terminal based onthe user operation in the second process, wherein the real object isdetermined to have the flat surface with the suitable region to displaythe virtual object based on an area of the suitable region being greaterthan an area threshold and an angle formed between a normal line of thesuitable region and a line-of-sight direction being less than an angularthreshold.